DynaCLR misc: CSV metadata sidecar, data-path profiling, DDP logging#485
Open
edyoshikun wants to merge 4 commits into
Open
DynaCLR misc: CSV metadata sidecar, data-path profiling, DDP logging#485edyoshikun wants to merge 4 commits into
edyoshikun wants to merge 4 commits into
Conversation
profile_stages.py records file bytes_read/read counts and chunk-cache reads after warmup so read amplification is measured directly rather than inferred from tensor shapes; profile_num_workers.py defaults to the Stage-1 self- positive recipe for a matched sweep. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
save_hyperparameters(ignore=...) so ContrastiveModule/BetaVaeModule checkpoints carry their config; add sync_dist to hparam logging and skip empty sample grids to avoid DDP logging desync. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add viscy_data.meta_csv (build_provenance_fields, metadata_to_rows, write_meta_rows_csv, read_meta_rows_csv) and thread a csv_dir option through the preprocessing path so normalization/QC statistics can be written to and read from per-store CSV sidecars instead of the zarr .zattrs, opening the store read-only. Datasets mounted without write access can then be prepared and trained on. Wires csv_dir through VisCyTrainer.preprocess, generate_normalization_metadata, the QC metrics run + config, the dynaclr cell-index preprocessing (CLI + Nextflow module + workflow) and its parquet reader, and airtable prepare (check_preprocessed + batch discovery). qc gains a viscy-data dependency for the shared module; uv.lock updated accordingly. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Catch-all branch for DynaCLR work that is independent of the requeue (#478) and rescale/LOT-correction (#424) PRs. Four atomic commits:
viscy_data.meta_csvmodule and acsv_diroption threaded through the preprocessing path (VisCyTrainer.preprocess,generate_normalization_metadata, QC metrics, dynaclr cell-index CLI + Nextflow module, the parquet reader, and airtable prepare). Lets datasets mounted read-only be prepared/trained by writing normalization/QC stats to per-store CSV sidecars instead of the zarr.zattrs.qcgains aviscy-datadep;uv.lockupdated (also drops the<3.14Python cap).save_hyperparameters(ignore=...),sync_diston hparam logging, skip empty sample grids.docs/clusters/reef.md, which is added on lightning config for requeing jobs on a preempted slurm partition #478; the link resolves once both merge intodynaclr_models.🤖 Generated with Claude Code